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Abstract 



Studies have been made on the phase transition phenomena of an oscilla- 
tor network model based on a standard Hebb learning rule like the Hopfield 
model. The relative phase informations — the in-phase and anti-phase, can 
be embedded in the network. By self-consistent signal-to-noise analysis (SC- 
SNA), it was found that the storage capacity is given by a c = 0.042, which 
is better than that of Cook's model. However, the retrieval quality is worse. 
In addition, an investigation was made into an acceleration effect caused by 
asymmetry of the phase dynamics. Finally, it was numerically shown that 
the storage capacity can be improved by modifying the shape of the coupling 
function. 

PACS numbers: 87.10.+e, 05.90.+m, 89.70,+c 
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I. INTRODUCTION 



Two hypotheses are available on the carrier of information in the brain. One, the rate 
coding hypothesis, states that information is represented by the density of spikes. The 
other, the temporal coding hypothesis, states that information is represented by the timing 
of neuronal firings, that is, the synchronization of oscillatory neural activities. 

Visual information is divided into a number of features, e.g., color, form, motion, and 
so on, which are processed in parallel in different areas of the brain, von der Malsburg |I[ 
has emphasized the necessity of temporal coding to bind these features, and have called 
into question conventional models based on rate coding. It should be noted that some 
physiological phenomena |§ have supported his discussion. Therefore, this discussion may 
also increase interest in temporal coding. 

One merit of temporal coding is that the information processing (e.g., optimization — 
minimization of an energy function) achieved by temporal coding neurons (i.e., coincidence 
detectors) is faster than that achieved by rate coding neurons (i.e., integrators) ||. 

Recently, therefore, the oscillator neural network has been attracting the attention of a 
growing number of researchers. 

In this paper, we discuss a class of oscillator neural networks that store two- value memory 
patterns into synchronously oscillating states. This class of oscillator neural networks is 
difficult to study analytically, and therefore was not discussed in previous works [§,§. To 
overcome the difficulty, we employ a powerful method |T2| that can be applied to various 



networks, and derive the memory capacity. Our model is more biologically relevant than 
previous models HH, since it is based on a real number synapse which is specified by a 
standard Hebb learning rule. 

In previous analyses fUljT]] anc ^ our analysis, a coupling function of a phase equation 
was approximately assumed as the sine function for mathematical tractability. In a large 
population of uniformly coupled oscillators with scattered natural frequencies (which cor- 
responds to ferromagnetic models), the behavior of the system is invariant to the shape of 
the coupling function, that is, the critical variance of the natural frequency causing a phase 
transition is only dependent on the lowest frequency components of the coupling function [|J . 
However, the former analyses do not support our analysis, since properties of the cross-talk 
noise in our system are different from those of scattered natural frequencies which break the 
mutual entrainment. 

In this parer, in order to verify effects of the shape of the coupling function on the 
performance of a network, we estimate the memory capacity of more realistic models. In 
addition, we numerically show that, in the case of an associative memory (on the basis of an 
oscillator network), gaps in the coupling function between in-phase and anti-phase improve 
the performance. 



II. ANALYSIS OF AN OSCILLATOR NETWORK BASED ON A STANDARD 

HEBB LEARNING RULE 

In general, when the coupling is sufficiently weak, the high- dimensional dynamics of a 
coupled oscillator system can be reduced to the following phase equation 0, 
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dfa ^ , 
at j=1 

where N is the total number of oscillators, fa is the phase of the ith oscillator and Jij 
denotes a synaptic weight. g{fa) is a periodic function. Figures |I](a)(b) show examples of 
g(<f)) values (sine-like-curves) obtained by the numerical calculation of weakly-coupled BVP 
(Bonhoeffer-Van der Pol) oscillators. 

For mathematical tractability, we approximately assume g{fa) = sin (</>). Then, Eq. [I] is 
expressed as, 



dt 



N 

JijSin^i - fa). (2) 



In Cook's model (a Q-state clock model with oo) 0, is a complex number synapse 
specified by a generalized Hebb learning rule |7],f|, which can not be easily achieved in bio- 
logical implementation. Cook's model can be regarded as an extension of Hopfield networks 
to "multi-states" . Here, is specified by the following learning rule, 

Jv = 7i't&tf> = exp«), (3) 

where {0f }i=i,...,N,n=i,-,p are phase patterns to be stored in the network, and are randomly 
assigned to or 7r with a probability of 1/2. 

We define a parameter a (loading rate) such that a = p/N. This learning rule is 
equivalent to a standard Hebb learning rule like in the Hopfield model ||, which can be 
easily achieved in biological implementation: increases between simultaneously firing 
cells; otherwise, decreases. 

In area CAl of the hippocampus, stimulation on the in-phase of the hippocampal theta 
rhythm includes a long-term potentiation that can be depontentiated by stimulation on the 
anti-phase of the theta rhythm ||10||. Thus, oscillator network models with the standard 



Hebb learning are simple and biologically plausible, yet have to be analyzed mathematically 



The system (0) has the following potential V: 

1 N V 

V = - oTf E E cos ((0,. - Bt) - (<p 3 - , (4) 

ij fi 

This potential (|]) is invariant under the transformation 0j — >• fa + (3 with any (3 G R. 
In other words, the equilibrium solution is irresistant to a uniform shift of fa (the neutral 
stability). Hence, the relative phase informations, i.e., the in-phase and anti-phase, can be 
embedded in the network. 

Here, we study the equilibrium properties of system (Q) when p, N — >• oo with fixed 
a = p/N, by using self-consistent signal-to-noise analysis (SCSNA) |12| which is based on 
the S/N analysis to explain the equilibrium properties. The results of applying SCSNA 
to the Hopfield model 0] and Cook's model coincide with those of the replica theory 



ilffl. SCSNA is a powerful method, since we can easily study the equilibrium properties 
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of analog-neuron networks without considering the potential, for example, analogue neural 



networks with asymmetric connections [12]. 

Defining the order parameters m% and (overlap) in a large N limit as, 



]yEefsin, 



we obtain the following equilibrium condition by setting d<pi/dt = in Eq. 0, 

cos (f>i = X (h[, h£) = h\ I \Jh\ 2 + h\ 2 , 

--Y(h\,hi 



Sllli 



hi 



h\ 2 + h} 2 



h' 



(5) 

(6) 
(7) 

(8) 



Note that we have ignored another solution cos< 



-X, sin . 



-Y by applying the 



so-called Maxwell rule [II] of statistical mechanics to find a relevant solution. Assuming 



m\,m\ = 0(1) and m^,m^ = 0(l/yN) for fi > 1, that is, £j as a condensed pattern, and 
expanding a polynomial around m%, = (// > 1) to split the cross-talk noise into an 
effective self-coupling and a Gaussian random variable |T2"| , |rT| ] , we obtain the following nine 
dimensional equations for the order parameters, 



m c = 


({X(x u 


^2))) , 


(9a) 


m s = 


({Y( Xl ,x 2 ))), 


(9b) 


Qc = 


((X 2 (X! 


^2))) , 


(9c) 


Qs = 




£2))) , 


(9d) 


Qsc = 


((X(xi,x 2 )Y(x 1 ,x 2 ))) , 


(9e) 


c x = 


1-A / 


[{QlXl + Q 3 a; 2)X(xi,X2))) , 


(9f) 


c 2 = 


1-A/ 


J(Q 3 xi + Q 2 x 2 )X( Xl ,x 2 ))) , 


(9g) 


Si = 




(Qi^i + Q 3 x 2 )Y(xi,x 2 )^ , 


(9h) 


S 2 = 


1-A, 


(Q3X1 + Q 2 ^ 2 )^(^l,^2))) , 


(9i) 



where ((•••)) is taken to mean the Gaussian average over X\,x 2 , that is, ((•••)) = 
/ / Dx\Dx 2 ■ ■ •, and the above equations satisfy C 2 = Si and q c + q s = 1. Note that 
gauge transformations were performed on variables of the condensed pattern. The pattern 
superscripts 1 of m c , m s are omitted for brevity. A and the Gaussian measure Dx\Dx 2 are 
expressed as follows, 



A = C x + S 2 + C 2 S 1 - C.S- 



2i 



Dx\Dx 2 



dx\dx 2 
27Tv/detQ 



cxp 



■-[x u x 2 ]Q 1 



Xi 



(10) 
(11) 



Q 

Q- 1 

Qi 
Q 2 



Qi Qs 
Q 3 Q2 
1 



detQ 

= (1- 
= S\q c 
= S X {1 
+ C 2 (1 



' Q2 


-Q 3 ~ 




Qi 


Q 3 ~ 


-Q 3 


Qi 




_Qs 


Q 2 _ 



S 2 yq c + 2C 2 (l~S 2 )q sc + Ciq s , 



2S x {l-C x )q sc - 
S 2 )q c + (l — 3 2 
- C x )q s . 



(1 " Ctfq., 
C x + S X C 2 + S 2 C x )q s 



(12) 

(13) 

(14) 
(15) 

(16) 



cos0 = X(x x ,x 2 ) and sin (ft = Y(x x ,x 2 ) at equilibrium satisfy the following condition, 
— ((1 — A)m c + \faxA sin + ((1 — A)m s + yfax^ cos 



a 



C 2 sin 2 — S x cos 2 + (C x — £2) sin cos < 



(17) 



In general, Eq. admits four solutions owing to the effective self-coupling terms cor- 
responding to the right-hand side of Eq. |17]. Figure ^| shows intersection points of the 
left-hand side and the right-hand side of Eq. [T7|, where the filled circles and open circles cor- 
respond to stable solutions and unstable solutions, respectively. Here, we find an available 
solution by applying the Maxwell rule O] in thermodynamics. As shown in Fig. ^|, there 
are two enclosed areas A x and A 2 , provided that the left-hand side of Eq. [L7| is to be the 
upper boundary and the right-hand side of Eq. [17| is to be the lower one. According to the 
Maxwell rule ||11|| , we must select a stable solution with the larger enclosed area. 

The above order parameters are analogous with those of the Hopfield model [13]. q c , 
q s , and q sc correspond to the so-called Edwards- Anderson order parameter, which measures 
the local ordering of XY-spins. C x , C 2 , S x , and S2 correspond to the susceptibility, which 
measures the sensitivity to external fields. 

The above equations for the order parameters are more complicated than those of Cook's 
model (three-dimensional equations) 0. Because patterns stored in Cook's model are uni- 
formly distributed in [0, 2ir], 2-dimensional Gaussian noise-Eq. [11] is isotropic, which enables 
us to reduce to three-dimensional equations in terms of the polar coordinate system. How- 
ever, in the case of our model, 2-dimensional Gaussian noise-Eq. [11] is anisotropic, since 
phase patterns stored in our model are only assigned to or 71". Thus, we can not reduce to 
simpler equations any more. 

Here, we define a new order parameter m = \Jrn? c + m 2 s for convenience. The solid curve 
in Fig. ^(a) shows the values of m for various values of loading rate a, which were obtained 
by solving the above equations numerically. The plots in Fig. |3](a) show results obtained by 
numerical simulation with = 2000. Note that the solution of the above equations forms 
an isotropic manifold as shown in Fig. 0(b), because system (0) is invariant to a uniform 
shift of 0j . Fig. |](a) shows a cross-section of this manifold. 

The theory is in good agreement with the results of this simulation. The storage capacity 
is given by a c = 0.042 and at this point the overlaps are m = 0.69. The storage capacity of 
our model is about 1/3 as large as that of the Hopfield model. In the case of Cook's model, 
a c = 0.038 and at this point, m = 0.899. Thus, the storage capacity of our model is better 
than that of Cook's model, but the retrieval quality is worse. 
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The dashed line in Fig. Q(a) denotes an unstable solution of the equations for the order 
parameters; m c = 1, m s = 0, q c — 1, q s — 0, q sc — 0, G\ — 0, C 2 = 0, S\ — 0, and S 2 = 1. 
The solution corresponds to the equilibrium solution fa = (perfect memory states) from 
Eq. 0. With a large N limit, the equilibrium solution fa = Qi l is unstable. 

Let us examine the stability of this solution at a finite N. Linearizing equation (0) 
around fa = 8}, we obtain the following matrix A: 

Jll — Y,j Jljdi) ^12^1^2 

•^21^2^1 ^22 — J2j J2j^j 

A = 

where we obtain eigenvalue with eigenvector r = [1, • • ■ , 1] T which corresponds to a uniform 
shift of fa, as previously discussed. We numerically calculated the maximum eigenvalue of 
A for various values of loading rate a. Figure ^ indicates the results, where (a) N = 10, (b) 
N = 100, (c) N = 1000 and (d) N = 2000. According to Fig. |, at p < 2, the system is 
neutrally stable around fa — @h an d at p > 2, the system is unstable, independently of the 
values of N. Thus, we can estimate that the capacity for perfect memory retrieval is given 
by p c = 2. 



J2N^N 
JNN — J2f JNitljZi 



(18) 



III. ACCELERATION EFFECT 

As shown in Figs. |l](a)(c), g{fa) obtained from real systems is asymmetric with respect 
to the origin, that is, g(fa) ^ —g(—<p). In this case, all phase values fa continue rotating with 
a uniform frequency, while keeping the relative states — in-phase and anti-phase. This phe- 
nomenon corresponds to the so-called acceleration effect |TJ]]. Let us consider the following 
perturbed system for Eq. 0, 

dfa N 

—j± = - J ij ( sin (<^ - 4>j + o)- sin a) . (19) 

When a is sufficiently small, we obtain the following equation from Eq. (|T 

dfa N N 

-jr = -^2 J ij sin (<^ - 4>j) ~ a S J v ( cos (^ ~~ fa) ~~ l ) ' ( 20 ) 

dt 3=1 3=1 

where the second term on the right-hand side is a structural perturbation — asymmetry of 
g(<f>). The neutral stability mode (a uniform shift of (pi) of Eq. § is broken by this structural 
perturbation. Solutions in the neighborhood of an equilibrium solution for an unperturbed 
system can be represented as, 

fa(t) = A + ujt + aui(r), (21) 

where r — at (slow time variable), and tpi denotes an equilibrium solution for the unper- 
turbed system, u represents a frequency of rotation caused by a structural perturbation. 
<JUi(r) is a higher-order fluctuation caused by the effects of a perturbation. Substituting Eq. 
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2T| into Eq. ^(| expanding a polynomial around o = and neglecting higher-order terms, 
we obtain 



w[i,---,i] T = g [ui, ■ ■ - ,u N y 



N 



N 



]T ( Clj - l) 

i=i 



G 



J\x — E, J\j c ij 



J12C12 



J21C2I J22 — J2f ^2jC2j 



, E J Nj ( c Nj ~ 1 

J\N C 1N 
J2NC2N 



JniCni 



JN2CN2 



J NN ~ E"f JNjCNj 



c i:j = cos(^ -lf)j), 



(22) 



(23) 



(24) 



where we obtain r = [1, • • • , 1] T G kerG which corresponds to a uniform shift of 0j. We can 
expect kerG = span{r}, since, if other modes with eigenvalue were to exist, the relative 
states would be broken by a perturbation. We can erase fluctuation term G [ui, ■ ■ ■ , ujA 



Ill 



Eq. 



taking an inner product between r and Eq. |22|. Thus, to can be expressed as, 

N 1 N p 



00 



^E^'-^EE cos O - at) - (^ - 0? 



(25) 



where the second term on the right-hand side is the potential V of the unperturbed system- 
Eq. |]. Therefore, when u 7^ 0, a stationary solution is transformed into a rotating solution 
by a structural perturbation — asymmetry of g(<f)). 



IV. MEMORY CAPACITY OF REALISTIC MODELS 

In order to verify effects of the shape of the coupling function on the memory capacity, we 
estimated the memory capacity of weakly-coupled BVP oscillators with diffusional couplings; 

TX{ = C {Xi Xi /3 -|- Ui) -\- £ Ej Jij ( r >ft\ 

Tin = -{xi + byi- a) /c + e Ej Jij (Vj - Vi) ' 
i = l,...,iV. 

When £ is sufficiently small, Eq. ^ can be reduced to the phase- variable description ([TJ). 
Figures |l](a)(b) show g((/>) values obtained by the numerical calculation, where (a) a = 0, 
6 = —0.5, c = 5.0, and (b) a = 0, b = 0, c = 5.0. We started the numerical calculation using 
phase equation ([j]) with raw g(fa) values, where N = 1000. 

As previously discussed, by the asymmetry of g((f>), all phase values fa went on rotating 
with a uniform frequency, while keeping the relative states — in-phase and anti-phase. By 
using the pattern overlap m = \/m c 2 + m s 2 , we can observe the macroscopic relative states, 
since (Ei & cos(ujt + fa)) 2 + (Ei & sin(u;r + <^)) 2 = (Ei&cos^) 2 + (Ei&sin^) 2 . 

Figures 0(a) (b) show final values of m versus a. The plots were obtained by numerical 
simulation. The solid curves were obtained by SCSNA (g(4>) = sin(0)). In Fig. 0(a), the 
theory is consistent with the results from the simulation. In Fig. 0(b), the memory capacity 
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is larger than the approximated system (Eq. |2|). These simulations showed that the gaps of 
g((j)) in Fig. |l](b) improved the performance of the network. 

BVP oscillators consist of an activator Xj and an inhibitor y^. The movement of the 
activator is rapid like a pulse wave, but that of the inhibitor is gentle like a sine wave. The 
gaps in g(<p) in Fig. |l|(b) were caused by a rapid phase- locking of the activator. 

We numerically estimated the memory capacity of the artificial model with the following 
g(<f>) values, 

qU) = { Sin ^' -f + 2A;vr < < f + 2A;vr, 
1 7sin(0), otherwise, 

k = • ■ • , —3, —2, —1, 0, 1, 2, 3, • • • , 

where 7 is the control parameters for the length of the gaps. Figures ^|(a)(b)(c)(d) show 
final values of m and the boundary of attraction versus a, where (a) 7 = 1.0, (b) 7 = 0.5, 
7 = 0.25, and (c) 7 = 0.05. In Fig. |6|(c), the memory capacity is larger than that of the 
original system (7 = 1.0), and the width of the basin is almost the same as that in Fig. |6](a). 
In Fig. 0(d), both the memory capacity and the width of the basin are worse than those in 
Fig. H(c). From these figures, we could guess that the optimal gap is given by 7 ~ 0.25. 



V. CONCLUSION 

Studies were made on the phase transitions of an oscillator neural network model based 
on a standard Hebb learning rule like in the Hopfield model. By SCSNA, it was found that 
the storage capacity is given by a c = 0.042. In addition, it was numerically shown that the 
gaps in the coupling function between in-phase and anti-phase improve the performance. 
This phenomenon show that a realistic model, i.e., the pulse-coupled oscillator has a good 
performance compared to a simple "XY-spin" . 

The storage capacity of the Hopfield model can be improved by replacing the usual mono- 



tonic output function with a nonmonotonic one ||15|| . The susceptibility becomes negative by 
the nonmonotonicity of the output function, so the variance of the cross-talk noise decreases. 
However, under the present conditions, we can not determine whether the properties of the 
gaps of the coupling function might be analogous to those of the nonmonotonicity in the 
Hopfield model. In future work, we need to expand our theory to allow it to deal with 
general g((p) values. 

In Cook's model and our model, only replica-symmetric solutions are considered. SCSNA 
is based on replica-symmetry, because the Gaussian ansatz for the cross-talk noise in SCSNA 
corresponds to the replica-symmetry ansatz in the replica theory. It is well-known that 
replica-symmetry breaking (RSB) |16| occurs in Cook's model, since the entropy of this 



system becomes a negative value [O. We can also guess that replica-symmetry breaking 
occurs in our model. Therefore, we need to derive RSB solutions |18[] of the oscillator 
network model as in the Hopfield model. However, the derivation of the RSB solutions is 
more difficult than with the Hopfield model. 

The author thanks Dr. K. Kurata and Dr. M. Okada for their valuable discussions. This 
work was partially supported by Grants-in-Aid for the Encouragement of Young Scientists 
No. 0782 and JSPS Research Fellowships for Young Scientists. 



S 



REFERENCES 



[1] For example, C. von der Malsburg, Am I thinking assemblies?, in Brain Theory 

(Springer- Verlag, 1984). 
[2] For example, A. K. Engel, P. Konig, A. K. Kreiter, T. B. Schillen, and W. Singer, 

Trends in Neuroscience, 15[6], 218 (1992). 
[3] J. J. Hopfield and A. V. M. Hertz, Proc. Natl. Acad. Sci. USA, 92, 6655 (1995). 
[4] H. Daido, Pro. of Theo. Phys., 89[4], 929 (1993). 

[5] G. B. Ermentrout, Journal of Mathematical Biology, 6, 327 (1981); Y. Kuramoto, Chem- 
ical oscillations, waves and turbulence (Springer- Verlag, 1984). 
[6] J. Cook, J. Phys. A: Math. Gen., 22, 2057 (1989). 
[7] T. Aoyagi and K. Kitano, Physical Review E, 55, 7424 (1997). 
[8] K. Okuda, (unpublished). 

[9] J. J. Hopfield, Proc. Natl. Acad. Sci. USA, 79, 2554 (1982). 
[10] C. Holscher, R. Anwyl, and M. J. Rowan, J. Neuroscience, 17[16], 6470 (1997). 
[11] T. Fukai and M. Shiino, Europhys. Lett., 26[9], 647 (1994); T. Fukai and M. Shiino, 

Neural Computation, 7, 529 (1995). 
[12] M. Shiino and T. Fukai, J. Phys. A: Math. Gen., 25, L375 (1992). 
[13] D. J. Amit, H. Gutfreund, and H. Sompolinsky, Physical Review Letters, 55[14], 1530 
(1985). 

[14] C. Meunier, Biol. Cybern., 67, 155 (1992); S. Mizuno and K. Kurata, Technical report 

of IEICE (in Japanese), NC94-15, 9 (1994). 
[15] M. Morita, Neural Networks, 6, 115 (1993). 

[16] J. R. L. de Almeida, and D. J. Thouless, J. Phys. A: Math. Gen., 11[5], 983 (1978). 

[17] T. Aoyagi, Private communication, (1996). 

[18] G. Parisi, J. Phys. A: Math. Gen., 13, L115 (1980). 



9 



FIGURES 



FIG. 1. (a)(b) Examples of g(4>) which are obtained by the numerical calculation of 
weakly-coupled BVP oscillators, i.e., Eq. |2(], where (a) a = 0,6 = — 0.5, c = 5.0, and (b) 
a = 0,6 = 0,c = 5.0. 

FIG. 2. A schematic diagram of the Maxwell rule to select relevant solutions to the equilibrium 
condition. 

FIG. 3. (a) Values of m versus loading rate a. The solid curve denotes a stable equilibrium so- 
lution, and the dashed line shows an unstable equilibrium solution which corresponds to (pi = Oj + P 
with any (3 £ R. These lines are obtained by SCSNA. The plots show results obtained by numerical 
simulation with N = 2000. (b) Manifold of the solution. Figure ||(a) shows a cross-section of this 
manifold. 

FIG. 4. The maximum eigenvalue of matrix A versus the loading rate a, where (a) N = 10, 
(b) N = 100, (c) N = 1000 and (d) N = 2000. At p < 2, the system is neutrally stable around 
4>i = 0} , and at p > 2, the system is unstable, independently of the values of N. 

FIG. 5. (a)(b) Values of m versus a. The plots are obtained by numerical calculation using 
phase equation (||) with the g((j)) values shown in Figs. 0(a) (b), respectively; N = 1000. The solid 
curves are obtained by SCSNA (g((j)) = sin(0)). 

FIG. 6. Basin of attraction and memory capacity of an artificial modal with controllable gaps. 
The upper data points represent the equilibrium overlap. The lower data points denote the bound- 
ary of attraction. All of the data is obtained by numerical simulation when N = 512 for five trials, 
except Fig. |(a) (N=1024). The error bars indicate standard deviations, (a) 7 = 1.0. (b) 7 = 0.5. 
(0)7 = 0.25. (d) 7 = 0.05. 
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